Thought recollection and speech assistance device

ABSTRACT

Some embodiments of the inventive subject matter include a method for detecting speech loss and supplying appropriate recollection data to the user. Such embodiments include detecting a speech stream from a user, converting the speech stream to text, storing the text, detecting an interruption to the speech stream, wherein the interruption to the speech stream indicates speech loss by the user, searching a catalog using the text as a search parameter to find relevant catalog data and, presenting the relevant catalog data to remind the user about the speech stream.

BACKGROUND

Embodiments of the inventive subject matter generally relate to thefield of voice and speech recognition software and its uses, and moreparticularly voice and speech recognition and assistance with speech.

Today, many electronic devices (e.g. mobile phones, tablets, computers,and televisions) can receive voice input from a user. Allowing voiceinput provides a user with a hands-free alternative to interact with thedevice. This is especially popular in the mobile phone industry. In themobile phone industry, voice input can be used to allow a user toprovide commands to a mobile phone without diverting their hands orattention from another task. Although there are currently many uses forboth voice and speech recognition software in mobile phone applications,a use does not exist which assists users that have trouble with speechloss. Speech loss can occur for many reasons, including disease orrecovery from disease, surgery, etc. One predominant cause of speechloss is Alzheimer's disease. Users suffering from Alzheimer's diseasemay have trouble completing sentences because they are unable toremember the information they wish to convey in their sentences in fulldetail. Likewise, user's suffering dementia may have a difficult timecompleting sentences because they are unable to remember the informationthey wish to convey in their sentences in full detail. Additionally,patients recovering from a stroke may likewise find it difficult tocommunicate effectively because of an inability to describe thoughts inwords. In addition to speech loss occurring due to disease, speech losscan also occur because of old age, poor memory, or even sleepiness.

SUMMARY

Some embodiments of the inventive subject matter may include a methodfor detecting speech loss and supplying appropriate recollection data tothe user. The method can include detecting a speech stream from a user.The method can include converting the speech stream to text. The methodcan include storing the text. The method can include detecting aninterruption to the speech stream, wherein the interruption to thespeech stream indicates speech loss by the user. The method can includesearching a catalog using the text as a search parameter to findrelevant catalog data. The method can include presenting the relevantcatalog data to remind the user about the speech stream.

BRIEF DESCRIPTION OF THE DRAWINGS

The present embodiments may be better understood, and numerous objects,features, and advantages made apparent to those skilled in the art byreferencing the accompanying drawings.

FIG. 1 a depicts a user 104 providing voice input to a speech-assistdevice 102.

FIG. 1 b is a diagrammatic illustration of one embodiment of aspeech-assist device 106 interface.

FIG. 2 is a block diagram of a speech-assist device 200.

FIG. 3 is a flow diagram illustrating operation for interpreting inputand presenting recollection data.

DESCRIPTION OF EMBODIMENT(S)

Some embodiments of the invention work with a computational device, suchas a mobile phone, to allow the “recollection” of a user's memory uponrecognizing a fault or delay in their speech. In one embodiment, thespeech-assist device may utilize both voice recognition and speechrecognition. The speech-assist device may utilize voice recognition tofocus only on the user's voice, disregarding errant voices andenvironmental din. The speech-assist device may utilize speechrecognition to convert the input speech into a text string. Thespeech-assist device may then store recent segments of text strings inmemory. Should the user pause for a significant period of time, orotherwise provide input indicative of speech loss, the speech-assistdevice may use the mobile device's internal search capabilities tosearch using all or a portion of the stored text as a search parameter.The speech-assist device may search through the user's contacts,photographs, current or past GPS locations, text messages, emails, etc.to find information relevant to the user's conversation. Upon findingrelevant information, the speech-assist device may present theinformation to the user, through visual or auditory cues, recollectingthe user's memory and allowing continuation of speech.

The description that follows includes exemplary systems, methods,techniques, instruction sequences and computer program products thatembody techniques of the present inventive subject matter. However, itis understood that the described embodiments may be practiced withoutthese specific details. For instance, although examples refer to thespeech-assist device as an application running on a mobile phone orother computing device, in other embodiments of the inventive subjectmatter the speech-assist device may run as a stand-alone unit on adedicated computing device. In other instances, well-known instructioninstances, protocols, structures and techniques, relating to voicerecognition and speech recognition specifically, have not been shown indetail in order not to obfuscate the description.

FIG. 1 a depicts a user 104 providing voice input to a speech-assistdevice 102. In FIG. 1 a, the user 104 is speaking in proximity of thespeech assist-device 102. In turn, the speech-assist device 102 isreceiving the user's 104 speech as input.

FIG. 1 b is a diagrammatic illustration of one embodiment of aspeech-assist device 106 interface. The speech-assist device 106comprises a computing device 108 and a method for presenting output (inthis embodiment a display) 110. In some embodiments, the display 110 maypresent to the user a textual display of the recent input 112. Upondetecting speech loss, the speech-assist device 106 may determine theappropriate recollection data. In some embodiments, based on appropriatecircumstances, the speech-assist device 106 may present recollectiontext 114 or other recollection data, such as a recollection picture 116.It is unnecessary that the speech assist device 106 include a display110. In some embodiments, the speech-assist device 106 may provideauditory recollection data output. In embodiments comprising a display110, recollection data may be presented as text, images, contactinformation, GPS information, etc.

FIG. 2 is a block diagram of a speech-assist device 200. Thespeech-assist device 200 includes a processor unit 202 (possiblyincluding multiple processors, multiple cores, multiple nodes, and/orimplementing multi-threading, etc.). The speech-assist device includesmemory 206. The memory 206 may be system memory (e.g., one or more ofcache, SRAM, DRAM, zero capacitor RAM, Twin Transistor RAM, eDRAM, EDORAM, DDR RAM, EEPROM, NRAM, RRAM, SONOS, PRAM, etc.) or any one or moreof the above already described possible realizations of machine-readablemedia. The memory 206 may include a voice cache 208, a voice to textconverter 210, a catalog search unit 212, a voice analyzer 214, and acatalog 216. The speech-assist device also includes a bus 204 (e.g.,PCI, ISA, PCI-Express, HyperTransport®,InfiniBand®, NuBus, etc.), anetwork interface 218 (e.g., an ATM interface, an Ethernet interface, aFrame Relay interface, SONET interface, wireless interface, etc.), astorage device(s) 220 (e.g., optical storage, magnetic storage, etc.),and an assignment management unit 226. The assignment management unit226 embodies functionality to implement embodiments described above. Theassignment management unit 226 may include one or more functionalitiesthat facilitate assignment of data elements in a data associationinterface. Although depicted as being coupled to the bus 204, theassignment management unit 226 may be embodied in the memory 206. Anyone of these functionalities may be partially (or entirely) implementedin hardware and/or on the processing unit 202. For example, thefunctionality may be implemented with an application specific integratedcircuit, in logic implemented in the processing unit 202, in aco-processor on a peripheral device or card, etc. Further, realizationsmay include fewer or additional components not illustrated in FIG. 2(e.g., video cards, audio cards, additional network interfaces,peripheral devices, etc.). The processor unit 202, the storage device(s)220, and the network interface 218 are coupled to the bus 204. Althoughillustrated as being coupled to the bus 204, the memory 206 may becoupled to the processor unit 202.

In FIG. 2, a speech stream from the user is captured as input by amicrophone 224. The speech stream is then stored in a voice cache 208within the main memory 206 of the speech-assist device 200, where thespeech stream is converted from audio to text by the voice-to-textconverter 210. Additionally, in some embodiments, the speech-assistdevice 200 may incorporate voice recognition technology. In suchembodiments, a voice analyzer 214 may determine whether the speechstream is the user's voice, or otherwise errant noise. Upon detectingspeech loss, the speech-assist device 200 may utilize a catalog searchunit 212 to search a catalog 216 of media recently accessed by the user.Upon finding relevant recollection data, the speech-assist device 200may then present the recollection data to the user via a mediapresentation device 222.

FIG. 3 is a flow diagram illustrating operation for interpreting inputand presenting recollection data. The operations of flow 300 begin atblock 302. At block 302, the speech-assist device receives voice inputfrom the user in the form of a speech stream. In some embodiments, thespeech-assist device may include a voice analyzer to utilize voicerecognition technology, which may allow the speech-assist device toaccept only the user's speech as input for the speech string. The flowcontinues at block 304.

At block 304, the speech-assist device converts the speech stream totext. In some embodiments, the speech-assist device may utilize speechrecognition technology to convert the speech stream into text. The flowcontinues at block 306.

At block 306, the speech-assist device stores the text. In someembodiments, the speech-assist device may store only the last ten tofifteen words of text from the speech stream. In other embodiments, thespeech-assist device may store more or less of the speech stream. Theflow continues at block 308.

At block 308, the speech-assist device determines if there is aninterruption to the speech stream indicating speech loss. In someembodiments, a long pause in the speech stream may indicate speech loss.In other embodiments, specific words or phrases can be utilized by theuser to indicate speech loss. In other embodiments, the speech-assistdevice may employ controls which may allow the user to indicate speechloss. Such controls may comprise buttons on the device that may allow auser to signal speech loss.

In some embodiments of the inventive subject matter, the speech-assistdevice can be “trained” by the user, setting a threshold pause lengththat will indicate speech loss. In other embodiments, the speech-assistdevice may be trained using the aforementioned controls. In suchembodiments, a user may indicate whether recollection data wasunnecessary when it was supplied, and likewise indicate whetherrecollection data was necessary when it was not supplied. If speech lossis detected, the flow continues at block 310.

At block 310, the speech-assist device searches the catalog using thetext as a search parameter. The flow continues at block 312.

At block 312, relevant catalog data is presented to the user. Relevantcatalog data may consist of any type of media. For example, the catalogdata may consist of a photograph, an email, a contact list, a textmessage, a recital of recent GPS locations, or information from thecalendar. In some instances, the relevant data may simply consist of theportion of the sentence before speech loss. In such instances, thespeech-assist device may simply display the last ten to fifteen wordsthat have been converted to text. The flow continues at block 314.

At block 314, the speech-assist device determines if the user's speechhas resumed. If the user's speech has not resumed, the flow returns toblock 310. If the user's speech has resumed, the flow continues at block316.

At block 316, if the user's input is complete, the process ends. If theuser's input is not complete, the flow returns to block 302.

As will be appreciated by one skilled in the art, aspects of the presentinventive subject matter may be embodied as a system, method or computerprogram product. Accordingly, aspects of the present inventive subjectmatter may take the form of an entirely hardware embodiment, an entirelysoftware embodiment (including firmware, resident software, micro-code,etc.) or an embodiment combining software and hardware aspects that mayall generally be referred to herein as a “circuit,” “module” or“system.” Furthermore, aspects of the present inventive subject mattermay take the form of a computer program product embodied in one or morecomputer readable medium(s) having computer readable program codeembodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent inventive subject matter may be written in any combination ofone or more programming languages, including an object orientedprogramming language such as Java, Smalltalk, C++ or the like andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The program codemay execute entirely on the user's computer, partly on the user'scomputer, as a stand-alone software package, partly on the user'scomputer and partly on a remote computer or entirely on the remotecomputer or server. In the latter scenario, the remote computer may beconnected to the user's computer through any type of network, includinga local area network (LAN) or a wide area network (WAN), or theconnection may be made to an external computer (for example, through theInternet using an Internet Service Provider).

Aspects of the present inventive subject matter are described withreference to flowchart illustrations and/or block diagrams of methods,apparatus (systems) and computer program products according toembodiments of the inventive subject matter. It will be understood thateach block of the flowchart illustrations and/or block diagrams, andcombinations of blocks in the flowchart illustrations and/or blockdiagrams, can be implemented by computer program instructions. Thesecomputer program instructions may be provided to a processor of ageneral purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions, which execute via the processor of the computer orother programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

While the embodiments are described with reference to variousimplementations and exploitations, it will be understood that theseembodiments are illustrative and that the scope of the inventive subjectmatter is not limited to them. Many variations, modifications,additions, and improvements are possible.

Plural instances may be provided for components, operations orstructures described herein as a single instance. Finally, boundariesbetween various components, operations and data stores are somewhatarbitrary, and particular operations are illustrated in the context ofspecific illustrative configurations. Other allocations of functionalityare envisioned and may fall within the scope of the inventive subjectmatter. In general, structures and functionality presented as separatecomponents in the exemplary configurations may be implemented as acombined structure or component. Similarly, structures and functionalitypresented as a single component may be implemented as separatecomponents. These and other variations, modifications, additions, andimprovements may fall within the scope of the inventive subject matter.

What is claimed is:
 1. A method comprising: detecting, via a microphoneof an electronic device, an audible speech stream; converting theaudible speech stream to text; detecting speech loss in the audiblespeech stream indicating an inability to complete a thought associatedwith the audible speech stream; generating a query in response to thedetecting the speech loss, wherein the query includes at least a portionof the text; searching a catalog on the electronic device using thequery to find related catalog data, wherein the catalog includes one ormore of text strings, images, contact information, and globalpositioning system information associated with the electronic device;and presenting the related catalog data.
 2. The method of claim 1,wherein the related catalog data is presented visually.
 3. The method ofclaim 1, wherein the related catalog data is presented audibly.
 4. Themethod of claim 1, wherein the speech loss is detected by a divergencefrom a normal speech pattern.
 5. The method of claim 1, wherein thespeech loss is detected via user input.
 6. The method of claim 1,wherein the detecting speech loss is trained via user training input. 7.The method of claim 1, wherein the speech loss is detected by a gap inspeech which exceeds a predetermined length of time.
 8. The method ofclaim 1, further comprising distinguishing a user's input from otherspeakers having conversations within the vicinity of the user, whereinthe audible speech stream includes the user's input.
 9. The method ofclaim 1, wherein the related catalog data is presented via a preferredmedia-type, the preferred media-type indicated via user input.
 10. Anapparatus for speech-assist, the apparatus comprising: a processor; anda computer readable storage medium having computer usable program codeembodied therewith, the computer usable program code executable by theprocessor to cause the apparatus to: detect, via a microphone of theapparatus, an audible speech stream; convert the audible speech streamto text; detect speech loss in the audible speech stream indicating aninability to complete a thought associated with the audible speechstream; generate a query in response to the detection of speech loss,wherein the query includes at least a portion of the text; search acatalog on the apparatus using the query to find related catalog data,wherein the catalog includes one or more of text strings, images,contact information, and global positioning system information recentlyaccessed via the apparatus; and present, on at least one mediapresentation device of the apparatus, the related catalog data.
 11. Theapparatus of claim 10, wherein the apparatus is configured to detect theaudible speech stream associated with a particular user.
 12. Theapparatus of claim 10, further comprising one or more input mechanismsoperable to indicate the speech loss in the audible speech stream. 13.The apparatus of claim 10, further comprising one or more inputmechanisms operable to train the apparatus to recognize the speech lossin the audible speech stream.
 14. The apparatus of claim 10, furthercomprising a display device to present the search results.
 15. Theapparatus of claim 10, further comprising an audio output device topresent the search results.
 16. A non-transitory computer programproduct for speech-assist, the computer program product comprising: acomputer readable storage medium having computer usable program codeembodied therewith, the computer usable program code comprising computerusable program code to: detect, via a microphone, an audible speechstream; convert the audible speech stream to text; detect speech loss inthe audible speech stream indicating an inability to complete a thoughtassociated with the audible speech stream; generate a query in responseto the detection of speech loss, wherein the query includes at least aportion of the text; search a local catalog using the query to findrelated catalog data, wherein the catalog includes one or more of textstrings, images, contact information, and global positioning systeminformation; and present the related catalog data.
 17. Thenon-transitory computer program product of claim 16, wherein the relatedcatalog data is presented visually.
 18. The non-transitory computerprogram product of claim 16, wherein the related catalog data ispresented audibly.